A Finite Time Analysis of Two Time-Scale Actor Critic Methods

Actor-critic (AC) methods have exhibited great empirical success compared with other reinforcement learning algorithms, where the actor uses the policy gradient to improve the learning policy and the critic uses temporal difference learning to estimate the policy gradient. Under the two time-scale learning rate schedule, the asymptotic convergence of AC has been well studied in the literature. However, the non-asymptotic convergence and finite sample complexity of actor-critic methods are largely open. In this work, we provide a non-asymptotic analysis for two time-scale actor-critic methods under non-i.i.d. setting. We prove that the actor-critic method is guaranteed to find a first-order stationary point (i.e., $\|\nabla J(\boldsymbol{\theta})\|_2^2 \le \epsilon$) of the non-concave performance function $J(\boldsymbol{\theta})$, with $\mathcal{\tilde{O}}(\epsilon^{-2.5})$ sample complexity. To the best of our knowledge, this is the first work providing finite-time analysis and sample complexity bound for two time-scale actor-critic methods.Comment: 45 page

Spin tunneling properties in mesoscopic magnets: effects of a magnetic field

The tunneling of a giant spin at excited levels is studied theoretically in mesoscopic magnets with a magnetic field at an arbitrary angle in the easy plane. Different structures of the tunneling barriers can be generated by the magnetocrystalline anisotropy, the magnitude and the orientation of the field. By calculating the nonvacuum instanton solution explicitly, we obtain the tunnel splittings and the tunneling rates for different angle ranges of the external magnetic field ($\theta_{H}=\pi/2$ and $\pi/2<\theta_{H}<\pi$). The temperature dependences of the decay rates are clearly shown for each case. It is found that the tunneling rate and the crossover temperature depend on the orientation of the external magnetic field. This feature can be tested with the use of existing experimental techniques.Comment: 27 pages, 4 figures, accepted by Euro. Phys. J.